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FOREWORD 



This research and development was performed under Work Unit E1176-PN-01 
(Improving the Navy's Computer-Managed Training System), as part of an R&D project to 
improve the Navy's operational computer-managed instruction (CMI) system, and 
was sponsored by the Deputy Chief of Naval Operations (OP-01). 

This report, the first of a series on Navy CMI, describes the problem areas that limit 
the effectiveness of the CMI system and the R&D proposals tliat have been developed to 
address these problems. Future reports will describe work being done under these 
proposals. Results will be used by the Chief of Naval Education and Training (CNET), 
Chief of Naval Technical Training (CNTT), commanding officers of all the Navy CMI 
schools, and others concerned with computer-based instruction. 

Appreciation is extended to Mr. Dwain Chambers (CNTT Code N-82ft) (CMI system 
manager), the itr.ff and students at all of the technical training schools on the' CMI 
system, and the fervice School Command, San Diego, for their assistance during the 
project. Also, appreciation is extended to Dr, H. S. Pennypacker, Jr., University of Florida, 
Gainesville, for the consulting service he provided during the problem analysis and initial 
R&D phases. 



3AMES F. KELLY, 3R. jAMES 3. REGAR 

Commanding Officer Technical Director 
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SUMMARY 

Background 

_ The Navy is currently operating a computer-managed instruction (CMl) <^v^f^m 
involving approximately 9000 students a day in 10 technical tracing schools As le 
^^Jf» students, courses, and capabilities, a coordinated R&D effort is required to 

» ensure the most cost-effective and productive technical training system. 

Objectives 

The objectives of this project were to identify the problem areas that limit the 
effectiveness of the CM I system and to develop coordinated R&D plans to solve problems 
amenaole to research ^lutions. 

Approach 

Major system problems were identified and suitable research proposals were devel- 
oped through a four-phase effort! f f 

PhaseJ. Problems were identified by observing the system and by administerinB 
. qu^tionnaircs to and holding structured interviews with CMI management, instruction 
stati, and student personnel. 

• Phase 11. Other computer-based instruct .n systems, both civilian and military, were 

examined to identify analogous problems and solutions. ■ 

P?!?^^ Candidate research proposals were developed to support the major problem 
areas identified, 

P^ase IV . Members of the research and operational technical training communities 
assessed the proposed research plans in terms of (1) importance of the problem involved^ 
UMeasibihty of the proposed research, (3) research and human resources requirements • 
and (if; expected costs and benefits of applying the product of the RdcD. 
Results 

The following six Rdf D proposals were given the highest priority by CNTT and CNET 
and form the basis of NAVPERSRANDCEN's initial CMI R&D efforts 

1. Effects of incentive charts on rate of progress through a CMI course. 

It 

2. Instructor role in a CMI environment. 

; 3. Computer-generated rpp -rts for the management of student learning. 

.^.J^ Develop- :orporation of automat ^fr tests into the CMI 

c I +• ^ °' alternate test strategies to Improve m atery and retention in 

selected CMI courses, 

6. Development of computer software to aid data summarization for research and 
management analysis. 
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INTRODUCTION 



Problem and BackBround 

Development of the Navy CM I System 

Since the 1960s, the Navy Personnel Research and Development Center (NAV- 
PERSRANDCEN) and Ite antecedent laboratories have been working with operational 
Navy training commands to develop cost-ettectivej computer-based training systems for 
large-scale operational training. In 1968, the Naval Personnel and Training Research 
Laboratory, San Diego (NPTRL) and the Chief of Naval Air Technical Training 
(CNATECHTRA) formally entered into m advanced development project to develop a 
computer-mwaged instruction (CMI) system. During 1972, the first course—Aviation 
Fundamentals- *was officially conducted with the prototype system developed under this 
project In 1974, CNATECHTRA and the Chief of Naval Education and Training (CNET) 
adopted CMI as a formal component of the Navy training system* In 1975| the Navy 
Education Training and Instructional Systems Activity (NETISA) (now the Management 
Biformation and Iristructional Systems Activity (MnSA)) let a contract for automated data 
proce^ing (ADP) hardware and software services for the eKtating system* Since that 
time, CNET and the Chief of Naval Technical Training (CNTT), witti NAVPERSRANDCEN 
support, have worked to develop courses and expand the CMI system. A comprehensive 
R&D project is needed to support the Navy CMI system as it expands In size and 
capabilities. 

Physical Characteristics of the Navy CMI System 

The Na'^ CMI system is one of the largest computer-based Instruction systems In 
existence. This system, using off -line, individualized Instruction materials, currently 
i ^ nages the daily Insfruction progress of about_9000 studente in 10 technical training 
schools at S locationsi Millington, TN| Orlando^ FL| Great Ukies, IL| San Diego, CA| and 
Pensacolat FL* These schools teach the following courses^ Basic Electricity and 
Elecfronics, Radioman "A," Propulsion Engineering Basics, Aviation Fundamentals, Avia- 
tion Mechanics, Avionics ^*As" and a portion of the ElectKwic Warfare Fundamentals 
course. The central computer facility for the system, which is located at MIISA, 
Millington, TNf grades tests, does record keeping, assigns study instructions, and provides 
information to instructors, school directors, and the CMI system manager (ranging from 
data ^out individual students to summaries of system effectiveness). The CMI system 
wjU eventually expand to a projected capacity of about 15,000 students in 25 schools in 
tocations. When fully o^raf ional, th^ system will ^ re^onslble for the Instruction 
management of nearly 30 percent of all the students In Navy technical training. 

Although the physical layoiH of the learning centers (LCs) In the Navy C ? system 
varies from school to schwl, the same components are present in all center §. u leralKs 
each LC consists of a room with rows of individual study carrels. Areas ior the 
distribution of ins^ ction materials and equipment are l^ated near each LC. Achieve- 
ment testing of ttr knowledge or theory portion of the course may occur at either the 
student carrels or L% a designated test area. In both cases, after the test is completed, 
the student submits a machine*scorable answer sheet Into an optical scanner (OPSCAN 
Model 17), and a teletypewriter printer (GE Termlnet Model 1200) provides feedback to 
the student in the form of a Learning Guide. In some CMI schools, special areas are set 
up for testing specific performance skills. For example, the students may troubleshoot 
equipment or components that have been prefaulted. ^e instructor in tach LC Is 
responsible for 25 to 30 students, who may stek assistance at any tlmw througr^f im 
training day. Thm total training day is S hours, with 6 hours typic^ iiv GSII 

instruction and 2 hours in other military training. To accommodate ^^ mb^ -t of 



students, some achools may operate two or even three consecutive shifts of trainees a 
day, obtaining maximum benefit from the physical plant and staff. Since individual 
instruction is given, and students work at varying paces, study continues during the entire 
6-hour shift. Students take breaks at their own discretion. 

Figures la through If illustrate a typical Navy CMI LC at the Basic Electricity and 
Electronics School, San Diego. 

The CM! Instruction Process 

Navy CMI focuses on student interactions with the curriculum materials (including 
text matter and equipment), the instructor, and the computer system. Figure 2 shows the 
possible interactions of system components during the instruction process. 

A student may begin the CMI course at any time, receiving a module study 
assignment to be worked at a self-determined pace. Each module generaJly takes 2 to 5 
houTS to complete. The student interacts with the curriculum materials by selecting and 
studying the various instruction media or pieces of equipment. Although most instruction 
material is printed text, sound slides and other media are also used. Students take a 
"progress check" test (usually self-scored) to determine whether they have mastered the 
lesson materials before they take the final module test. 

When the students feel that they have mastered all objectives for the module, they 
take a module knowledge test from a test center. Responses are entered on a machine- 
readable answer sheet, which is inserted into an optical scanner. Responses are then 
processed by the scanner and transmitted on leased telephone lines to the central 
computer at Millington, TN. Within 30-60 seconds, the computer identifies the student 
and the test, scores the test, stores responses In a student history file, determines the 
next assignment, and transmits test results and the next assignment back to the student. 
A typewritten Learning Guide, obtained from the printer near the scanner (Figure Ic), 
indicates missed ftuestions, ists lessons or objectives needing additional study, and 
informs the student of remediation tests that he must complete after such study. After 
the student has mastered all module objectives, the next module is assigned. 

On the existing CMI system, the Instructor transmits performance test responses to 
the computer by completing a check sheet and making an administrative entry. A more 
automated procedure is currently being followed at the Radioman "A" school, where 
automated teletyping performance testing has been developed. Here, a direct connection 
between the student response (teletypewriter keyboard performance) and the computer 
provides automatic evaluation of student performance. 

Students interact with the instructor for a variety of reasons. For example, they may 
ask questions about course subject matter, request additional materials, interact for 
personal reasons (e.g., problems in or out of school that affect progress), or simply request 
confirmation of their progress. To interact well with students, instructors must have 
technical competence, teaching ability, and interpersonal counselling skills. They must 
handle administrative and subject-matter queries, know the Instruction materials and 
equipment, and be able to interpret and use the student progress information available 
from the computer and school. This computerized progress information comes from o 
Student Response History file, which includes individual student records, or from a dally 
Learning Roster, which indicates the current state of the Instructional progress of all 
students in the instructor's LC. The roster tells the instructor what modules students are 
studying, fpw long they have been assigr.'ed specific modules, and, most important, how 
the actual instructional progress of each student compares with his predicted progress, 
based 




Figure 2. A diagram of thm relmtion of components of the Navy CMI system, 

on his Armed Services Vocational Aptitude Battery (ASVAB) test score?, age, and school 
records. Ideally^ this actual versus .predicted student progress Information would enable 
the instructor to take the corrective action nepded to assure satisfactory progress of 
every student through the course. Althoi*:>h self-pacing occurs through the instructional 
materials, it is expected that the student wiH^rfiaintain a ::tudy rate that will result ^^ 
course completion within the predicted time, and that the instructor will a^lst the 
. student in maintaining a minimum predicted rate of progress. These instruction progress 
expectations minimise training time and facilitate timely assignment of school graduates 
to follow-on schools or fleet billets, 

-m 

Despite the current success of the system in managing student progressj projected 
expansion dictaf:es the following questions! 

It Can improved instruction Interactions bring more cost-effective training? 

2. Can the system expand in size and capability without excessive cost growth? 

3. Can improvements in Navy CM! apply to other services and to industry and 
academia? 

Objectives 

The objectives of this effort were to identify the problem areas that limit the 
effectiveness of the CMI system and to develop coordinated R&D to solve problems that 
are amenable to research solutions. 
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APPROACH 



The project commenced Ln 1977 and was conducted in the following four phasesj 

1. Identification of CMI system problems and requirements. 

2. Review of other CMI systems. 

3. Development of candidate R&D proposals, 

4. Setting R&D priorities. 

These phases are described in the foilowing paragraphs. 
Phase I— Identification of CMI Syster n Problems 

The CMI system manager (CNTT Code N-g2«) Is responsible for maintaining an 
operational training system for thousands of students. Since the system manager is the 
focal point for complaints on systern pfoblems, he was able to identify a number of 
problems that he felt might be amenable to an R&D solution. Other CNTT staff 
members, representing training-program coordinators and CNTT research organization 
were asked to give their perception of CMI problems. 

User perceptions of CMI problems were obtai administering questionnaires on 

system effectiveness to the students, instructor. id training officers at the eight 
schools then on the CMI system. All elements of tne CMI process and. the procedures 
followed by the training schools were examined. Results showed that user perceptions of 
problems were consistent with those of the CNTT staff. These problems were primarily 
related to the interactions of system components— the student, the curriculum, t;«^ 
msfructor— and to the requirement to maintain a flow of properly trained students from 
course to course and out to the fleet. 

Thirteen specific problem areas were Identified and rJasslfied Into three gene-ral 
categories: (1) physical and environmental constraints, (2) student and system 
deficiencies, and (3) system capability limitations. Problems in physical and 
environmental constraints were most easily observed, but were not usually correctable by 
research. For example, repair of defective heating and cooling systems does not call for 



The second category— student and system deficiencies- -was characterized by unique 
problems students meet as they progress through the course. These difficulties stem from 
inadequate direction by either the computer or Instructor, or result from problems met In 
mastering the course materials or tests. 

The third genera] class of problems concerned the CMI system's limitations. For 
example, the computer system could not adequately maintain a student's step-by-step 
prc^gress record that was easily retrievable. Thus, If the student had difficulty in latter 
stages of the course, suitable records of past progress were not readily available to assist 
111 assigning remedial instruction. Failure to provide adequate learning within a course is 
most serious when a student who has completed the CMI course cannot demonstrate, 
during a follow-on course or the on-the-job, skills he is supposed to have acquired. 

Phase 11— Review of Other CMI Systems 

?n the second phase, large-scale CMI training systems maintained by the other 
military services, industry, and academia were reviewed to identify possible solutions for 
the types of problems identified In ihe Navy CMI system. A comprehensive review of alj 
computer-based systems was not attempted. Rather, the review focused on those systems 
designed to manage large numbers of students through individual Instruiction materials. 

ErJc 7 9 



Systems selected for review inclu^^ed :he Advanced Instructional System (AIS), at Lowry 
Air Force Basf, CO| tre Computerized TraiFUng System CCTS) at Fort Gordon, GA| and 
civilian systeff^s at the Universities of Illinois, Pittsburgh, California %t Irvim?, and 
Florida, Results of the reviews showed that virtually all of these systems expericinced 
similiir problems, Solutioni were mt directly available and would have to be tailored to 
specific systems. The Navy system, it was noted, was more advanced in some operational 
respects than many of the non-Navy systems. Certainly, the goals of military training 
made the Navy system more capable than non-Navy systems in predicting student 
progress»-desplt€ the problems associated with this feature of Navy CML 

Phase ni-^Development of Candidate R&D Proposals 

In the third phase, NAVPERSRANDCEN researchers and consultants, with input from 
the schools and management organizations, developed candidate R&D proposals in support 
of the 13 problem areas identified in Phase L These plans, which are listed below, 
described the problem, the proposed resaarch approach, and expected costs and benefits 
should the res^iarch be successful. 

L Expanding strategies and tactics for remediation in the Navy CMI system, 

2, Evaluating and monitoring knowledge and skill retention within and beyond the 
Navy CMI system, 

3. Development and application of student perception and motivation reports* 
Effects of incentive charts on rate of progress through a CMI course. 

5* Study management-system Implementation in m existing course of instruction, 

6. Instructor role in a CMI environment. 

7. The allocation of instructors in a CMI environment, 

8. Descriptive and predictive measurement of performance in the Navy CMI 
systems. 

9. Computer-gF.nerated aids for the management of student learning. 

10. Development and Incorporation of automated pp^^ ormance testing into the CMI 
system, 

11. Effect of test-item response characteristics on mastery and retention in 
selected CMI courses. 

12. Opportunities for software improvements in the Navy CMI system, 

13. Exploiting computer technology to improve computer-bfised instructional 
management. 

Pha^ IV--^ttinR R&D Priorities 

In the fourth phase, military and civilian researchers and members of the operational 
training and management organizations met to discuss the severity of the problems, 
feasibility of the proposed research solution, availability of reiearch and human resources, 



and costs and benefits of the solutions. After each proposal was presented and discussed, 
participants completed an open-ended auestionnaire to provide a summary of opinions. An 
abstract of mch proposal and a consensus on the importance of each are provided in the 
appendix^ 

After all 13 proposals had been discu&nQd, a final period was devoted to omissions, 
priorities, and the sequencing of the various R&D elements. One omission v/as identifl-d* 
the *velopment of a resource^allocation system to fac-M?ate the alignment of studants 
and instructors to study carrels, study materials, test equipment, test settings^ and 
learning centers. It was suggested that a program was needed to determine how tc get 
maximum value from minimum resources. Development of this resoarce-allocadon 
system was judged to be an operational rather than a research issue. 



RESULTS 

CM! Research Priorities 

The CNTT staff assigned priority to five R&D proposals that warranted immc late 
attention. CNET (Code N-5) endorsed CNTT's priorities and added another proposal to 
the list. Therefore, NAVPERSRANDCEN*s initial CMI research was directed toward the 
following six proposalsi 

1. Effects of inceativa charts on rate of progress through a CMI course. 

2. Instructor role in a CMI environment, 

3. Computer-generated aids for the management of student learning. 

4* Development and incorporation of automated performance testing into the CMI 
system. 

5. Development of alternate test strategies to improve mastery and retention in 
selected CMI courses (modification of Plan #11). 

6. Development of computer loftware improvements to aid data summaries for 
research and management analyiis (modification of Plan #12)* 

Initial Re search Efforts 

This section describes the initial research efforts conducted on the six R&D proposals 
receiving priority status. 

Effects of Incentive Charts on Rate of Prograss Through a CMI Course 

Since itudent motivation is a fundamental problem in CMI, some kind of incentive 
system is needed to prod students Into continuous activity. In this prograr^^ Incentive 
charts were used to record progress made, encourage study, and reduce training time. 

The lncentive_ chart experiment was conducted at irhe BE&E "A" school, Millington, 
TN, using a NAVPERSRANDCEN IBM 5110 computer ana civilian researchers. The results 
demonstrated a significant decrease in training time. The experimental group, which 
received the computer-generated incentive charts on requist, completed the course in 95 
hours, compared to 103 hours for the "no chart" control grcup. 
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If the procedu res work as well wi'^h the operational CM I equipment and Navy 
personnel as they did in the experimental test, the system can be used at CM I schools by 
merely "turning on'* the charts* An operational test Is planned with Navy instructors, 
multiple schools, and the Navy's centraTcomputer system. This project is an example of 
an R&D effort that has the potential to yield significant benefits with little implementa- 
tion costs. If the experiment results are repeated, a full training day per student could be 
saved with a cost avoidance of over $1 million annually. 

Instructor Role in a CMI Environment 

A contract has been let to McDonneiUDouglas, Inc. to develop the Ideal CMI 
instructor role-model for the Navy, Air Force, and a proposed Marine Corps CMI systems* 
Because of its cross-service application, this research was funded jointly by NAVPERS- 
RANDCEN and the Advanced Research Projects Agenr/. The functions defined for the 
instruction will be validated empirically in the Navy CMI system. Results of this project 
are expected to be available for the instructor training course being developed ty the 
Naval Education and Training Support Center, Norfolk, 

Tins research represents the first attempt to develop a CMI instructor role from a 
theoretical perspective followed by an empirical validation. Results from this research 
will impact on virtually every existing military and civilian CMI system.. Completion of 
the work is expected in FY 1 SSI. 

Computer-Generated Aids for the Management of Student Learning 

Work in this area has focused on improving the system's capability to predict a 
student's completion time* Although the system now predicts total course completion 
time fairly accurately, the prediction times for completion of individual modules must be 
improved. Work on prediction capability has focused on the BE&E course. 

Development and Incorporation of Automated Performance Testing into the CMI 
System 

This completed effort improved the efficiency of performance-skill training by using 
CMI to score and analyze performance tests automatically. Tele typing instruction In the 
Radioman "A" school at San Diego was chosen for this study because manual scoring 
procedures were tedious, time consuming, and subject to errors, feme instructors spent 
all day manually scoring typing tests. Personnel from the computer and instructional 
development organizatlons--the Management Information and Instructional Systems 
Activity (MIISA), and the Instructional Program Development Center (IPDC), San Diego, 
with support from NAVPERSRANDCEN, developed a system for automated testing of 
teietyping performance,- NAVPERSRANDCEN developed a system for providing diagnostic 
feedback to the student on the types of errors being made. With automated testing 
procedures, the average time required for students to complete a typical three-test series 
was cut from ^0 minutes to 2ft minutes. More importantly, the new system reduced 
typing-course completion time and student attrition rates. These savings are attributed 
to the Error Distribution Report (EDR) provided by the new system, which gives the 
student detailed information on areas where Improvement is needed. During the research, 
a daily EDR reduced course completion time from an average of 85 hours to 6^ hours, a 
savings of almost 3 full training days. The EDR also helped reduce attrition rates from 25 
to 10 percent in the experimental samples. The automated testing system and resulting 
EDRs are now in regular use at the Radioman school. 



EKLC 



Development of Alternate Test Strategies to Im p rove Mastery and Retention in C MI 
Courses ^ " — — — 



This research is concerned with determining the types of knowledge test items and 
testing procedures that are appropriate for technical training under computer manage- 
ment. At present, the CMI system is limited to using multiple-^choice questions* Answers 
to these test items are fed Into an optical scanner via paper answer sheet. Costs for 
answer sheets and optical scanners are very high. While 1:he multiple-choice testing 
format is convenient for machine scoring, its exclusive use may not result in the 
necessary level of mastery, since it is simply a measure of recognitioni In developing 
instructions, fill-in questions for testing recall of material must be used to satisfy certain 
types of training objectives. 

In response to such criticisms, the Propulsion Engineering (PE) school. Great Lakes, 
developed a method of converting fill-in item responses to a modified multiple-choice 
format that permitted machine scoring. This conversion procedure, however, may have 
decreased test accuracy and increased course completion time. During the research, 
'^'^^^H^^''^^'^'^^^'^ found that answer conversion did not decrease test scoring accuracy 
appreciably, but did require an average of an extra half day of training In a 22-day course. 
While all question types resulted In similar en d-of -^course achievement, fill-^in questions 
without cues resulted in better 2-week retention. 

Current work on alternate test strategies involves implementing some of these 
findings by eliminating the optical scanners and paper answer sheets. This approach would ^ 
reduce operating costs while improving the instructional capabilities of the system. 

Development of Computer Software Improvements to Aid Data Summaries for 
Research and Management Analysis 

This work has involved software to improve the system's management and research 
capabilities. The system must Improve its ability to use massive amounts of data used to 
track students and evaluate instructional materials and system effectiveness. 

Numerous improvements have already been made In the system's ability to summarize 
individual student, group, and other data to facilitate changes In manuals or procedures. 
The data summaries are available to school, management, or research organizations and * 
have been applied by CNTT and MIISA to the operational system. 

Future Research Issues 

Since the Navy CMI system has matured since the initial research priorities were set, 
and since part of the original R&D has been completed, the operational training 
organization priorities have shifted to meet current concerns. 

A CMI study conducted during the problem analysis compared alternate student to 
instructor (S/I) ratios in a CMI LC at the BE&E school, San Diego. The study investigated 
whether student performance would suffer if the S/I ratio were increased. The results 
suggested that, although the Increased S/I ratio did not result in performance decrements, 
training time increased in some career paths. The data revealed that the role of the 
instructor most ^c^efinitely affects student performance, and suggested that altered 
instructor duties may ultimately lead to more cost-effective training. 

From the S/I ratio study, it was apparent that a more comprehensive investigation 
was needed to determine the most cost-effective ratio. Further investigation should 

■ V ■ ■ ■ ■ . 
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involve different courses, larger sample sizesi a wider range of S/I ratios, and other 
variables, since different courses and schools will undoubtedly make different demands on 
their instructors. 



, From the test item study performed at the PE school, two research topics have 
emerged: (1) planning and evaluation of a test-answer input device (TID) that would 
replace the optical scanner and the paper answer sheets, and (2) development of proper 
GMI testing and remediation strategies (an extension of current research). , 

In addition to research essential to improving current CMI systems, planning for a 
future-generation Navy computer-based training system is needed. It is obvious that this 
planning must begin soon^ even if a major system change is not to occur for several years. 
This planning must include the economic analysis suggested by Orlansky and String^ to 
ensure development of a more cost effective system. It Is only by pressing the research 
questions and the available instructional technology that the CMI system can continue to 
handle its full share of the Navy training responsibility. 



^Orlansky, J,, & String, J, Cost-effectiveness of computer-based instruction in 
military training (IDA Paper P-1375J. Arlington, VAi Institute for Defense Analyses, 
1979, 7 f 
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CANDIDATE CMI R&D PROPOSALSj 
ABSTRACTS AND PLANNING SESSION CONSENSUS 

^...^'''♦k^PP^M,'^^''^'®^? abstracts of the 13 candidate research proposals developed 
during the CMI R&D Planning Session, along with a particlpait coniensus of the 
importance of each plan. Order of presentation does not imply ordefof ir^JortanL 



1. 



Expanding Strategies and Tactics for Remediation In the Navy GMI System 



_ Abstract. This project considers the development of remediation approaches 
applicable to the CMI system to improve basic skills of trainees, remedy skill deficiencies 
withm a course of instruction, or remedy technical skill deficiencies that occur across a 
sequence of CMI courses. T" 

Consensus. Although remediation of basic skill deficiencies is important, 
techniques to help acquire technical skills have even higher priority. Research should 
locus on the identification and use of remediation techniques that are the greatest aid to 
learning. Appropriate remediation techniques would simplify control of student learning 
progress during a course and would aid the development of individualized CMI course 
materials. . 

2* Evaluating and Monitoring Knowledge and Skill Re tention Within and Beyond 
the NavyCMI jystem [ ~ ' — - 

Abstract. The intent of this project is to determine how long students retain 
the knowledge and skills taught in the CMI courses. Their ability to retain knowledge will 
be assessed after they have begun work in the fleet or Just before or during follow-on 
training courses. Assessing knowledge and skill retention during a follow-on tralninp 
course will be aided by computer- managed testing programs that reflect the course-model 
sequencing and structure of the original course. Intracourse retention would be 
determined through similar computerized testing and an off-line assessment procedure. 

Consensus. The quantification of the retention problem is of great importance 
to the Navy, but must be handled appropriately to ensure comparison with proper baseline 
data when determining the extent of the problem in both CMI and non-CM I learning 
situations. Extensive system design effort may be needed for proper measurement of 
retention along with "fleet enthusiasm" for post-training assessment. The product of this 
RacD effort would have tangible residual benefits for operational training, since the 
programming would remain in the computer system itself. 

3' Developme nt and Application of Student Perception and Motivation Rep orts 

Abstract. The purposes of this project are to develop automated techniques for 
determining student perceptions of the instructional environment and to assess student 
motivation. The automated student critiques would enable schools and training manage- 
ment to obtain summarized data, routinely, on levels of student motivation and on student 
opinions of the quality of instruction. 

> 

Consensus. There is need for a service-wide effort that would facilitate the 
systematic collection of subjective data from the students about their school experiences. 
It should be possible to relate these data to changes made In the school. Use of student 
time in the administration of the critiques should be minimized, however, and the 
automated critiques should focus on benefits to the school and not on the acquisition of 
basic research data. • 



Effects of Incentive Charts on Rate of Progress Through a CM! Course 



Abstract. This R&D involves development of graphic displays of student 
progress in a CM I course as an incentive to study. Other incentives to study would be 
devised by school management to reinforce the motivation- provided by the displays. 

Consensus . This project is of sound design and important for the proper 
management of a CM I system. The effort would yield an additional and innovative means 
of measuring and motivating study progress. RacD costs would be low. System-wide cost 
could be estimated fairly accurately following the research effort. Previous research 
indicates that payoff from this R5cD effort would be high. 

Study-Management System Implementation In an Existing Course of Instruction 

Abstract. Cornputer-managed study techniques, developed in prior research 
efforts, would be Implemented in an existing lecture-based course. The R&D Intent would 
be to determine the feasibility of a low-cost method for individualizing a cf urse focusing 
on Navy schools that are not presently scheduled for full-scale instructional development. 

Consensus . This project serves the clearly worthwhile function Df developing 
cost-effective methods for individualizing additional courses into a CM I system. 
Selection of the course and careful course analyses and development are extremely 
critical to this effort. Accurate documentation of implementation activities and costs is 
necessary to derive maximum benefit from the RkD effort. 

6. Instructor Role in a CM I Envi' - onment 

Abstract . This project broadly describes the R&D necessary to determine the 
instructor's role in CMI. Specifically, the proposal calls for the specification of CMI 
instructor activities, development of an Interim in-service training program for immedi- 
ate use in CMI schools, and development of a formal training program for candidate 
instructors. A critical aspect of this proposal is the accurate specification of the 
instructor's role so instructor aids from other R&D efforts can be exploited. 

Consensus . Development of an instructor training program that accurately 
reflects the instructor duties in a CMI school is very important for a large operational 
computer-baied training system. It is clear that research Is needed for proper instructor 
role definition, since the majority of previous research has dealt with instructors in a 
lecture-based school environment, A vitally important aspect of the research is the 
proper use of Instructor evaluation meaaures. Measures of Instructor-effectiveness should 
reflect not only direct observation of Instructor activities, but also the degree of student 
success,, This research is particularly important in view of the limitations placed on 
initial selection of individuals for instructor duty. This project should be broadened to 
include the development of a training program for CMI school managers. 

7. Allocation of Instructors in a Computer-Managed Instructional Environment 

Abstract . This project will I'etermine the best CMI student-to-instructor ratio 
and learning-center configuration, V «ieks to determine the ratio bandwidth through 
which managerial perrogatives can vary wf ore cost or benefit thresholds are exceeded. 

Consensus . It is important to establish the S/I bandwidth so that management 
can make decisions on manning, but it is also necessary to consider the types of activities 
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assigned to instructors in the learning center. This research must be coordinated with 
Proposal 6, which involves overall instructor training. 

' Descriptive and P redictive Measurement of Performa nce in the Navy CMI 
Systems " ~ — — • — — — - 

Abstract. This project involves the development, test, and application of a 
technique, using test-response latency, to measure the student's mastery of course 
material. Data on how long students take to master segments of a course can be recorded 
during actual training. These data can be built into the computer to aid predictions if it is 
determined that the benefits will justify the cost. 

Consensus. This project will develop more sensitive measurement of mastery in 
a CMI course. Time is certainly one of the achievement variables. Although there is 
minimum R&D effort, the cost of applying the results of the R&D to the actual operating 
system may not be justified by the benefits. Cost-elfectiveness must be determined by 
accurate estimates during the research phase. The use of response latencies in measuring 
performance is important, since the measure of 100 percent mastery based solely on 
number of correct tast items is tqo insensitive for use in an operational training system. 

9* Computer-Generated Aids for the Measurement of Student Learning 

Abstract. This project examines the daily progress reports instructors receive 
in the CMI learning center. Both the predictive capabilities of the regression equation*: 
used m estimating student progress and the format for displaying information to tlie 
instructor require improvements. 

Consensus. This project has high priority because of an existing problem In all 
Navy CMI schools. Consideration should be given to expanding the project to include 
improvement of the data displays for the school and system managers. 

^0. Development and Incorporation of Automated Performance Test ing into Com- 
puter-Managed Instrucflon ^— 

Abstract. This project is intended to identify, automate, and incorporate in the 
CMI system those performance skill-testing activities that would result in more cost- . 
effective training and more accurate performance measurement. The initial effort in this 
project involves the automation of telety|.)ewriter-keyboard performance testing in the 
RM "A" school at .San Diego. The R&D will emphasize techniques that will facilitate 
automated CMI performance testing and will accelerate skill acquisition. 

Consensus . While the general Issue of identifying skills and automating their 
perfo^t^ance testing on a CMI system is important and has high' priority, care should be 
taken in the case of the RM school not to redo extensive typing training research. 

11. Effect of Test-Item Response Characteristics on Mastery and Retentio n in 
Selected Comput er-Managed Courses ' ~~ ~ "~ — ^ 

Abstract . This project will determine test-Item formats compatible with CMI 
that provide maximum instruction and measurement benefi,ts. This research will examine 
the test response characteristics, along with other test item formats, currently in 
question at the Propulsion Engineering school at Great takes. The validity of each test 
response procedure will be assessed against an independent performance test. 
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Consensus . Because the proposal addresses a broad and recurring probJem in 
the existing test system used with Navy CMI, the R&D effort should move beyond the 
limitations imposed by simple 5-alternative multiple-choice questions. The proposal will 
apply to all CMI schools, although the focus is on test response characteristics for 
inri mediate use. Clearly, the validity of any of the response formats should be carefully 
determined for the particular subject matter being taught. This projiect may well be the 
source of some of the most practical innovations to enrich and extend the existing Navy 
CMI system. 

i2. Opportunjtles for Software Improyements to the Navy CMI System 

Abstract . This project is designed to determine fully the users and uses of the 
data available on the Navy CMI system. The data use determination would focus on, those 
uses that could be exploited by the existing computer system In the near future. Users 
would include not only the personnel from the schools (students, instructors, and 
managers), but also the training system managers, computer scientists, and research 
personnel. Identified data uses would be refined and arranged in priorities with NETISAj 
the activity responsible for system analysis and programming. 

Consensus . This R&D effort will be useful school, management, and research 
personnelj successful completion is a necessary step toward full-scale use of the results of 
other research. The project Is clearly related to data-management problems currently 
experienced by the Navy and other CMI systems, and will be of real benefit at little cost. 
NETISA personnel should be intimately involved with this effort from the outset to 
maximize system benefits. 

13' Exploiting Computer Technology to Improve Computer-Based Instruction 
Management ' ' ~ 

Abstract . This R&D will determine the configuration of a second-generation 
computer system for Navy CMI. 

Consensus . This effort is a necessary R&D activity that will facilitate future 
operational decisions. While the effort Is very worthwhile, txperimental approaches 
besides the Delphi technique might be considered. R&D resources will be required, and 
the Navy will benefit greatly, although the benefit will be evident only when the planning 
for a subsequent computer configuration is initiated, A secondary benefit will be the 
establishment of a precedent and procedure for long-range planning In the broad area of 
military technical training. 
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